Methods for feature extraction

We have discussed some methods for feature extraction such as 2D, 2.5D, and 3D convolutions.

A convolution is an operation that applies a convolutional filter to each pixel of an image. A more detailed explanation could be found in this great Wikipedia article.

A simple 2D convolution with a 3x3 kernel could be visualized by the following animation:

We have just discussed several types of convolutions that could be used for feature extraction of 3D medical images:

2D Convolution is an operation visualized in the image above, where a convolutional filter is applied to a single 2D image. Applying a 2D convolution approach to a 3D medical image would mean applying it to every single slice of the image. A neural network can be constructed to either process slices one at a time, or to stack such convolutions into a stack of 2D feature maps. Such an approach is fastest of all and uses least memory, but fails to use any information about the topology of the image in the 3rd dimension.

2.5D Convolution is an approach where 2D convolutions are applied independently to areas around each voxel (either in neighboring planes or in orthogonal planes) and their results are summed up to form a 2D feature map. Such an approach leverages some 3-dimensional information.

3D Convolution is an approach where the convolutional kernel is 3 dimensional and thus combines information from all 3 dimensions into the feature map. This approach leverages the 3-dimensional nature of the image, but uses the most memory and compute resources.

Understanding these is essential to being able to put together efficient deep neural networks where convolutions together with downsampling are used to extract higher-order semantic features from the image.

Next up, we will take a closer look at how convolutions operate by running through a notebook and performing an exercise.